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ABSTRACT 



Ethnography may be used to inform the audience of a 
curriculum program with the activities of the educators and students 
involved. Thus the expository function of evaluation is best met by 
ethnography, A conceptual model developed at the Central Midwestern 
Regional Educational Laboratory incorporates ethnographic data and 
techniques into curriculum evaluation. The model uses three 
dimensions: data, including scale measures, questionnaire responses, 
and participant observation; the role of the evaluation — which is 
either to evaluate a completed program or to evaluate a program in 
progress so that the results of evaluation may be used to modify the 
program, which is then re-evaluated; and, the focus of investigation, 
which is on student, mediator, or material. The mediator is usually 
the classroom teacher. The model establishes the parameters of 
evaluation, and specifies and integrates three different approaches 
to evaluation. The ethnographic data may also serve as a source of 
new hypotheses and substantive theories, (JH) 
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The purpose of this presentation is quite simple: to explore with 

you the utilization of ethnography in curriculum evaluation. The subject 
is, i think, entirely appropriate both in terms of the nature of this 
symposium and, more generally, in terms of the current interest in 
evaluation at large. ^ 

In the brief amount of time allotted, I would like to touch lightly 
on the following issues: (l) some current thinking about the technology 

of evaluation; (2) a rationale for including ethnology in that newly 
emerging technology; (3) a brief description of an initial evaluation 
model which incorporates ethnology; (4) the utilization of that model 
in two recent CEMREL projects; and (5) some observations on the heuristic 
values of ethnology in evaluation studies. 

In the last few years there has been an increasing awareness that 
current curriculum evaluation technologies are both conceptually and 
operationally inadequate for the task at hand. Robert Stake (1967), 
in his introduction to Perspectives of Curriculum Evaluation , states 
bluntly, "New techniques of observation and judgment need to be 
developed. In fact, we need a new technology of educational evalua- 
tion. (p. 3)." A few paragraphs later he suggests what this new 
technology might be. "Our guess," he says, "is that this technology 
will draw from instructional technology, psychometric testing, social- 
survey technology, communications technology, and others (p. 4)." 

While Stake does not explicitly mention ethnography, it is implicit 
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in both his stress on "techniques of observstion" snd his subsequent 
statement of the purpose of evaluation. Again he writes: 

The purpose of educational evaluation is expository: 
to acquaint the audience with the workings of certain 
educators and their learners ... A ful 1 [his 
italics] evaluation results in a story, supported 
perhaps by statistics and profiles. It reveals per- 
ceptions and judgments that different groups and 
individuals hold--obtai ned, I hope, by objective 
means. As a bonus, it may offer generalizations 
("The moral of the story is . . .") for the guidance 
of subsequent educational programs, (p. 5) 

If this is an acceptable statement of purpose, then f believe the educa- 
tional ethnographers can make a substantial contribution to curriculum 
evaluation. By training and inclination the ethnologist is in a unique 
position to tell a "story." His field of vision is wider than the 
psychometrician's; his contacts— informants if you will— are likely to 
provide data more valid than that obtained via questionnaire or a brief 
interview; his extensive, time-consuming field work, whether in Micro- 
nesia, BlackfIsh Village, or Appalachia, may, either serendipical ly or 
in Malinowski's "foreshadowed problems" sense, reveal significant but 
overlooked variables at work and thus provide insights into the nature 
of a given curriculum that more traditional methodologies might well 
miss. This is a large claim, and one that many would dispute. None- 
theless, I think it is a claim that can be substantiated. Perhaps a 
few excerpts from the field notes taken one morning during our Computer 
Assisted Instruction (CAI) program might indicate how this claim can be 
justified. 

8:30 Arrived at the school about 8:30. Stopped into the school 
office and asked the secretary if she would make a list for 
me of the boys and girls in grades 1-6 .. . 

8:32 Dashed up to the teletype room. The machines were all clatter- 
ing away at great speed, but Miss Jane [system monitor] was not 
there. 

8:33 Went down to the opposite end of the building to visit the 

second grade class. On the way I met the third grade teacher 
and we chatted briefly . . . Miss. Jane came by as we talked 
to get the first graders . . . 

8:36-9:32 Observed second grade math class . . . 

9:32“9:^0 Chatted with teacher. 

9:^0 I went back up to the teletype room to watch the second graders 
as they were working on the teletypes. [Long list of names, 
drill numbers, percent correct, and times follow.] 
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Some observations on the actions of the second graders: 

I noted particularly that Billy H. was very nervous. As 
the teletype would type out the problem he would literally 
bang his hand on the side of the teletype. He was constantly 
in motion. 

• • • • 

Sam and Morgan were consistent finger counters 

• • • • 

Gail was a constant verbal izer. On first seeing a problem 
like 3 + 6 » 1 + ^, her comment was, "Oh, no. I can't do 
these." Then she verbalized the problem as "3+6=9 take 
away 1 is 8." 

• • • • 

Two other chatterboxes were Ricky and Ellen. Part of their 

constant chatter went as follows: 

Ricky: You and me go crazy on these computers. 

Ellen: I love to do these. 

Ricky: I love this, Ellen. 

Ellen: Me too. 

• • • * 

Talked a bit with Miss Jane re the daily report and what 
happened to it. She told me they were "filed away." 



10:30 Observed third grade math lesson. 



11:09 Back in the teletype area. The machines are not running due 
to system malfunction. Miss Jane has just received a phone 
call from one of the other schools that indicates the system 
has not been running for the last 25 minutes . . . She has 
had to restart the machine four times this morning .... 
Looked at one child's printout. Many machine errors. Child 
was quite shaken. 



11:58-1:05 Observed fifth grade math class. 

1:08 Checked teletype area. No one there. Checked for list 
with school secretary. Not ready. Left school. 

Even this brief account indicates the rich and varied data available 
to the ethnologi cal ly oriented evaluator. The three classroom visits 
afforded an opportunity to observe how and to what degree the CAI program 
was integrated into the normal mathematics lesson. Talking with the 
teachers gave some insight into their feelings about the program. Watch- 
ing the children at the teletypes revealed psychological aspects of the 
program, such as emotionality and anxiety, as well as some social dimensions. 
"Hard" data on student performance in terms of time taken to complete a 
drill and percentage scores were obtained. Some indices of student 



attitudes were also obtained. Certain administrative .'nd operational 
aspects of the program were observed, e.g., the filing of the daily 
report, the communications network between schools, the internal sched- 
uling, equipment breakdowns, and the like. All of this and more provides 
the raw materials for that "full story"— the plot, the setting, the 
characters, and the point of view-- that Stake calls for in a "full" 
evaluation. In short, I would argue that ethnology can make a contri- 
bution both to the emerging science of evaluation, and if the ethnolo- 
gist's account "reads like a novel," as some have reacted to one of our 
recent efforts, it might well be taken as a compliment! 



Yet the ethnologist, at least in Stake's view of things, is not a 
solo performer in the new technology of evaluation. Few investigators 
are equipped to carry out simultaneously the multiple facets of a "full" 
evaluation. Even granted the ability of a given individual to perform 
such a task, the magnitude of most large scale evaluations prohibits 
this. Clearly, what is needed is a conceptual model, or models, of eval- 
uation (Scriven, 1967), the technologies to be utilized, the focus of 
investigation, and the like. The initial outlines of such a model have 
been developed at the Central Midwestern Regional Educational Laboratory 
(CEMREL) under the leadership of Howard Russell and Louis Smith. While 
it was developed specifically for CEMREL 's Computer Assisted Instruction 
(CAI) evaluation, it seems to have a good deal of general izabi 1 i ty. 

This model is shown in Figure 1 . 



FOCUS 








Figure 1 The CEMREL Evaluation Model 
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Time does not permit even a moderate discussion of the model now. Let 
me say simply that it has both recognizable weaknesses and virtues. 
Clearly, it establishes the parameters of the evaluation. Even more 
important in Russell's words; 

Perhaps the most significant feature of the CEMREL 
Evaluation Model is the separation and integration of 
the three different perspectives in evaluating CAI . 

Where the perspectives converge on a situation and pro- 
duce data that are mutually reinforcing, the resulting 
conclusions have increased credibility and increased 
validity. In cases where the differing perspectives 
converge on the same situation and the resulting data 
are mutually contradictory, there is even greater 
value in the results. Under such circumstances, 
apparently valid conclusions are seen to be invalid 
and apparently credible statements are seen to be not 
so credible. It is under such circumstances that the 
knowledge base increases or becomes firmer as a result 
of the evaluation activities. (I969, p. 11 ) 

Perhaps two brief examples from the CAI evaluation will illustrate 
the potency of Russell's remarks, and at the same time illustrate the 
vitality of ethnology in such studies. In the overall design, provision 
was made for pupil, parent, and teacher attitude inventories as they re- 
lated to CAI. Without going into detail, one of the findings was that in 
general teacher favorabi 1 i ty toward CAI decreased over time. As nonpar- 
ticipant observers, our own frequent contacts with teachers corroborated 
this. In this sense we were obtaining "converging perspectives" and data 
that was "mutually reinforcing." In addition, however, we were able to 
document the more subtle nuances of change, the reasons for change, and 
the intensity of feeling. Moreover, in our own free-wheeling style, we 
were able to obtain simil >r though frequently contradictory attitude 
indices from a wider sam;: e of respondents: principals, superintendents. 

State Department officials. Title III agency personnel, and the like. This 
is to say again — that the ethnologist may gain a more comprehensive view 
of the totality of a project, a perspective that may be highly important 
for school administrators contemplating the adoption of a particular 
innovation. 

A second example revolves around the "hard" data amassed through 
achievement tests. Again, without going Into detail, the basic experi- 
mental design was a sophisticated pretest-tr( itment-posttest design uti- 
lizing randomly selected experimental and control groups within classes 
plus secondary control groups from classes where CAI was not used at all. 

In brief, while the results of the testing showed significant and positive 
effects on classes as a whole that utilized CAI, no significant differences 
were found between the experimental and control groups within classes. As 
non-participant observers we were able to offer at least a partial expla- 
nation for this somewhat surprising and disappointing finding. Our Field 




Notes from classroom observations indicated clearly that in many instances 
"mediators," in this case teachers primarily, were confounding the design 
by using the CAl printouts for whole class instructional purposes, per- 
mitting a manual containing the drill exercises to be used freely by all 
students as a supplementary workbook, by publicly displaying printouts 
in the classrooms, and the like. However, in addition to providing data 
vt/hich clarified the results of the achievement tests, our data shed light 
on a larger and more general research issue; namely, the extent to which 
control can be exercised over teacher behavior in experimental classrooms. 

All this is not to say that the ethnologist's role in evaluation 
studies is legitimate only to the extent that it provides supportive, con- 
tradictory, or explanatory assistance to the more customary methodologies. 
On the contrary, it has its own independent function of raising issues 
essential to that "full story" that other technologies are ill-equipped to 
handle. Again, in our CAl study, as nonparticipant observers, we were 
able to document the impact of such variables as the cultural setting, 
political maneuvering, precarious funding, decision-making processes, 
tecbnical and administrative complexities, and the like. It seems to us 
that if evaluation is "the discovery of the nature and worth of something 
(Stake S Denny, 1969, p. 370)," all these are issues of considerable 
importance, particularly to potential adopters. 

While the CAl evaluation was summative in nature, our current 
involvement with CEMREL's Comprehensive School Mathematics Program (CSMP) 
is formative. This is a new experience for us, and there are few guide- 
1 i nes to follow. 

The basic evaluation design follows the model previously depicted in 
Figure 1. A CSMP staff member serves as "in-house" evaluator, concentra- 
ting primarily on psychometrics. Dr. Smith and I are again engaged in 
participant observation, and in the absence of a survey specialist, all 
three of us have gathered questionnaire data. 

One of the areas of particular interest to us in this project is the 
exploration of the dynamics of ethnography in formative evaluation. We 
are concerned about such issues as the nature and form of feedback appro- 
priate to a development project, the timing of feedback, the personnel to 
whom it should be addressed, and, most important, exploring ways to avoid 
the ever-present dangers of cooptation. While the threat of cooptation 
lurks in the background of many extended investigations, it poses special 
problems for the ethnographer who in a fundamental way deoends upon close 
personal relationships with those whom he works. 

In our current work with CSMP we have adopted the following general 
procedures: (l) field work, that is, intensive classroom observations of 

the pilot trials, discussions with teachers, writers, and other staff per- 
sonnel, and attendance at staff meetings; (2) on the basis of such field 
observations, writing a series of memoranda on selected critical issues 
and submitting them to the appropriate CSMP staff members for their con- 
sideration; (3) meeting formally with the CSMP staff at a subsequent date 



to discuss issues raised in the memos. A concrete example might help to 
make this procedure clearer. 

One of the stated goals of the CSMP project is: "To develop suitable 
in-service and pre-service teacher training programs in connection with 
the individualized [CSMP] curricula (Kaufman 6 Ste iner, I969, p. 319).." 

The initial step in accomplishing this goal was a two-week summer workshop 
for instructional personnel who would be "piloting" the CSMP materials in 
their classrooms during the 1969-70 school year. As participant observers 
we attended the sessions, took part in the activities, and became acquainted 
with the materials and personnel. Following the workshop we prepared a 
report (Pohland £ Smith, 1969) and submitted it to CSMP. The contents of 
that report can be inferred reasonably well from these excerpts from the 
introduction: 

The third aspect of the plan of the paper is the development 
of a mechanism for formative evaluation proceeding from a partic- 
ipant observer stance. V/e present a brief account of the summer 
workshop, a 'descriptive narrative' based upon our field notes. 

As much as possible, this will minimize conceptual analyses, will 
summarize any comparisons to implicit or explicit criteria, and 
will minimize suggestions for future actions. In this way a 
concrete image of the program will he developed. Later, we will 
make less tentative analytic and evaluative comments. 

Then, we plan to back a’way from the CSMP workshop ana imple- 
ment a next step in a conception of formative evaluation. V/e 
propose first of all to look at some of the extant literature 
on workshops. In doing so, we will be particularly concerned 
with three issues: (l) how are workshops uniquely defined as 

types of in-service training programs; (2) what are the principal 
components that have been identified; and (3) what is the theo- 
retical rationale underlying workshops. Such a conceptual analy- 
sis should generate a number of ideas for reconsidering the prior 
descriptive data. 

Next, we wish to describe and analyze several workshops in 
which we have been involved as participants or as observers. 

Using the conceptual stance developed earlier on these concrete 
materials should enhance the generation and evaluation of alter- 
native approaches available to cujriculum and teaching workshops 
such as the one generated by CSMP. While this contains some 
hazards, we think it contains a most important element of 'con- 
structive' formative evaluation. 

Since most of the workshops to which we have addressed our- 
selves have included a major instructional component, we digress 
briefly to outline a number of instructional strategies which 
seem relevant for specific purposes and situations. 
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Finally, we plan to reconceptual I ze workshops and, in the 
process, develop a social systems model applicable to workshops, 
which, in conclusion, will contain some comments of the nature 
of temporary systems. 

Subsequent to submitting the report, we met with key CSMP staff members 
for discussion. As the conversation proceeded, as future plans and alter- 
natives were raised, discussed, modified, accepted or rejected, it seemed 
to us that we were mutually engaged in formative evaluation in the best 
sense. 

A fifth issue that I would like to raise is what might be called the 
heuristic value of participant observation. As we have argued elsewhere 
(Smith S Pohland, 1969), our preference is to use the raw observational 
data as the source of hypotheses. These we attempt to link into axio- 
matic chains (Zetterberg, 1965) ^nd, in effect, construct miniature 
substantive theories. Such propositional theories can then be tested by 
laboratory or field experiments.^ 

Again let me draw on the CAi study for an illustration of what might 
be done. As we observed the children working at the teletypes, talked 
with them, and listened to them talking among themselves, we became aware 
of the intensely competitive behavior that the program generated. The 
elements leading to competitiveness seemed to be threefold: the number 

of drills completed in a day, the score, and the time taken. These three 
elements were found singly or in combination. For example, one of many 
similar episodes we recorded in our Field Notes thus: 

Guy, L 6O2OA5, 100^ in 62 seconds. He talks to Karl re 
his time and indicates, "You gotta beat it." Karl, 

L 602035, groans with his first error. (5/29) 

We further noticed that while this type of behavior was visible in widely 
dispersed areas of the country, yet it seemed to be localized in particu- 
lar schools within a region and still more narrowly, only in certain 
classrooms within a school. 

On the basis of these observations a number of tentative research 
possibilities are suggested. One might be a series of experimental studies 
in which, on the basis of carefully formulated hypotheses, the time and 
score variables might be manipulated in order to assess their relative 
competition invoking weights. A second series of social psychological 
studies might be initiated to discover which variables are responsible 
for the high competitiveness in some settings but not in others. A 



^An excellent example of this research chain is the Smith & Kleine 
(1969) Teacher Awareness study which tested hypotheses generated earlier 
by Smith 6 Geoffrey 0^68) in their study of an urban classroom. 




third series might be designed to assess the impact of a competition 
evoking curriculum on personality development. 

We would urge this type of follow-up study for two reasons. First, 
it seems reasonable to. us that from the results of such studies curriculum 
developers would be in a more knowledgeable position to make decisions. 
Secondly, if the ethnographer can do more than simply tell a story and 
use his data to construct testable "grounded theory" in the Glaser and 
Strauss (1967) sense, then his work assumes theoretical as well as 
practical significance. In so doing, he can counter Scriven's (1967) 
Indictment: 

It is difficult to avoid the conclusion, however, that most 
process research of this kind In education, as in psycho- 
therapy (though apparently not in medicine), is fruitful 
at neither the theoretical nor the applied level, (p. 50) 

That remains an everpresent and continuing challenge to our efforts. 
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